Diarization using textual and audio speaker labeling

ABSTRACT

Systems and methods of diarization using linguistic labeling include receiving a set of diarized textual transcripts. A least one heuristic is automatedly applied to the diarized textual transcripts to select transcripts likely to be associated with an identified group of speakers. The selected transcripts are analyzed to create at least one linguistic model. The linguistic model is applied to transcripted audio data to label a portion of the transcripted audio data as having been spoken by the identified group of speakers. Still further embodiments of diarization using linguistic labeling may serve to label agent speech and customer speech in a recorded and transcripted customer service interaction.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority of U.S. Pat. No. 10,134,401,issued Nov. 20, 2018, which application claims priority of U.S.Provisional Patent Application Nos. 61/729,064, filed on Nov. 21, 2012,and 61/729,067 filed Nov. 21, 2012, the contents of which areincorporated herein by reference in their entireties.

BACKGROUND

The present disclosure is related to the field of automatedtranscription. More specifically, the present disclosure is related todiarization using linguistic labeling.

Speech transcription and speech analytics of audio data may be enhancedby a process of diarization wherein audio data that contains multiplespeakers is separated into segments of audio data typically to a singlespeaker. While speaker separation in diarization facilitates latertranscription and/or speech analytics, further identification ordiscrimination between the identified speakers can further facilitatethese processes by enabling the association of further context andinformation in later transcription and speech analytics processesspecific to an identified speaker.

Systems and methods as disclosed herein present solutions to improvediarization using linguistic models to identify and label at least onespeaker separated from the audio data.

BRIEF DISCLOSURE

An embodiment of a method of diarization of audio data includesreceiving a set of diarized textual transcripts. At least one heuristicis automatedly applied to the diarized textual transcripts to selecttranscripts likely to be associated with an identified group ofspeakers. The selected transcripts are analyzed to create at least onelinguistic model. A linguistic model is applied to transcripted audiodata to label a portion of the transcripted audio data as having beenspoken by the identified group of speakers.

An exemplary embodiment of a method of diarization of audio data from acustomer service interaction between at least an agent and a customerincludes receiving a set of diarized textual transcripts of customerservice interactions between at least an agent and a customer. Thediarized textual transcripts are group in pluralities compromising atleast a transcript associated to the agent and a transcript associatedto the customer. At least one heuristic is automatedly applied to thediarized textual transcripts to select at least one of the transcriptsin each plurality as being associated to the agent. The selectedtranscripts are analyzed to create at least one linguistic model. Alinguistic model is applied to transcripted audio data to label aportion of the transcripted audio data as having been spoken by theagent.

Exemplarily embodiment of a system for diarization and labeling of audiodata includes a database comprising a plurality of audio files. Atranscription server transcribes and diarizes the audio files of theplurality of audio files into a plurality of groups comprising at leasttwo diarized textual transcripts. A processor automatedly applies atleast one heuristic to the diarized textual transcripts to select atleast one of the transcripts in each group as being associated to anidentified group of speakers and analyze the selected transcripts tocreate at least one linguistic model indicative of the identified groupof speakers. An audio source provides new transcripted audio data to theprocessor. The processor applies the linguistic model to thetranscripted audio data to label a portion of the transcripted audiodata as being associated with the identified group of speakers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart that depicts an embodiment of a method ofdiarization.

FIG. 2 is a flow chart that depicts an embodiment of creating and usingan agent linguistic model.

FIG. 3 is a system diagram of an exemplary embodiment of a system ofdiarization of audio files.

DETAILED DISCLOSURE

Speech, transcription and speech analytics of an audio stream areenhanced by diarization wherein a speaker identity is identified andassociated with speech segments. A speaker diarization system and methodis aimed at identifying the speakers in a given call and associatingeach speech segment with an identified speaker.

Embodiments of a diarization process disclosed herein include a firststep of a speech-to-text transcription of an audio file to be diarized.Next, a “blind” diarization of the audio file is performed. The audiofile is exemplarily a .WAV file. The blind diarization receives theaudio file and optionally an information file from the speech-to-texttranscription that includes at least a partial transcription of theaudio file as inputs. Each audio segment or term in the information fileis associated between speakers based upon identified acoustic or textualfeatures. This diarization is characterized as “blind” as thediarization is performed prior to an identification of the speakers. Inan exemplary embodiment of a customer service call, the “blind”diarization may only identify speakers while it may still beundetermined which speaker is the agent and which speaker is thecustomer.

The blind diarization is followed by an agent diarization wherein anagent model that represents the speech and/or information content of theanent speaker is compared to the identified speech segments associatedwith the separated speakers. Through this comparison, one speaker can beidentified as an agent, while the other speaker is identified as thecustomer. One way in which one speaker can be identified as an agent isby linguistically modeling the agent side of a conversation, andcomparatively using this model to identify segments of the transcriptionattributed to the agent.

The identification of segments attributed to a single speaker in anaudio file, such as an audio stream or recording (e.g. telephone callthat contains speech) can facilitate increased accuracy intranscription, diarization, speaker adaption, and/or speech analytics ofthe audio file. An initial transcription, exemplarily from a fastspeech-to-text engine, can be used to more accurately identify speechsegments in an audio file, such as an audio stream or recording,resulting in more accurate diarization and/or speech adaptation. In someembodiments, the transcription may be optimized for speed rather thanaccuracy.

FIGS. 1 and 2 are flow charts that respectively depict exemplaryembodiments of method 100 of diarization and a method 200 of creatingand using an a linguistic model. FIG. 3 is a system diagram of anexemplary embodiment of a system 300 for creating and using a linguisticmodel. The system 300 is generally a computing system that includes aprocessing system 306, storage system 304, software 302, communicationinterface 308 and a user interface 310. The processing system 306 loadsand executes software 302 from the storage system 304, including asoftware module 330. When executed by the computing system 300, softwaremodule 330 directs the processing system 306 to operate as described inherein in further detail in accordance with the method 100 andalternatively the method 200.

Although the computing system 300 as depicted in. FIG. 3 includes onesoftware module in the present example, it should be understood that oneor more modules could provide the same operation. Similarly, whiledescription as provided herein refers to a computing system 300 and aprocessing system 306, it is to be recognized that implementations ofsuch systems can be performed using one or more processors, which may becommunicatively connected, and such implementations are considered to bewithin the scope of the description.

The processing system 306 can comprise a microprocessor and othercircuitry that retrieves and executes software 302 from storage system304. Processing system 306 can be implemented within a single processingdevice hut can also be distributed across multiple processing devices orsub-systems that cooperate in existing program instructions. Examples ofprocessing system 306 include general purpose central processing units,application specific processors, and logic devices, as well as any othertype of processing device, combinations of processing devices, orvariations thereof.

The storage system 304 can comprise any storage media readable byprocessing system 306, and capable of storing software 302. The storagesystem 304 can include volatile and non-volatile, removable andnon-removable media implemented in any method or technology for storageof information, such as computer readable instructions, data structures,program modules, or other data. Storage system 304 can be implemented asa single storage device but may also be implemented across multiplestorage devices or sub-systems. Storage system 304 can further includeadditional elements, such as a controller capable of communicating withthe processing system 306.

Examples of storage media include random access memory, read onlymemory, magnetic discs, optical discs, flash memory, virtual memory, andnon-virtual memory, magnetic sets, magnetic tape, magnetic disc storageor other magnetic storage devices, or any other medium which can be usedto storage the desired information and that may be accessed by aninstruction execution system, as well as any combination or variationthereof, or any other type of storage medium. In some implementations,the storage media can be a non-transitory storage media. In someimplementations, at least a portion of the storage media may betransitory. It should be understood that in no case is the storage mediaa propogated signal.

User interface 310 can include a mouse, a keyboard, a voice inputdevice, a touch input device for receiving a gesture from a user, amotion input device for detecting non-touch gestures and other motionsby a user, and other comparable input devices and associated processingelements capable of receiving user input from a user. Output devicessuch as a video display or graphical display can display an interfacefurther associated with embodiments of the system and method asdisclosed herein. Speakers, printers, haptic devices and other types ofoutput devices may also be included in the user interface 310.

As described in further detail herein, the computing system 200 receivesan audio file 320. The audio file 320 may be an audio recording or aconversation, which may exemplarily be between two speakers, althoughthe audio recording may be any of a variety of other audio records,including multiple speakers, a single speaker, or an automated orrecorded auditory message. In still further embodiments, the audio filemay be streaming audio data received in real time or near-real time bythe computing system 300.

FIG. 1 is a flow chart that depicts an embodiment of a method ofdiarization 100. Audio data 102 is an audio recording of a conversationexemplarily between two or more speakers. The audio data may exemplarilybe a .WAV file, but may also be other types of audio or video formats,for example pulse code modulated (PCM) format and linear pulse codemodulated (LPCM) audio files. Furthermore, the audio data is exemplarilya mono audio file; however, it is recognized that embodiments of themethod disclosed herein may also be used with stereo audio files. Onefeature of the method disclosed herein is that speaker separation indiarization can be achieved in mono audio files where stove speakerseparation techniques are not available.

In embodiments, the audio data 102 further comprises, or is associatedto, metadata 108. The metadata 108 can exemplarily include dataindicative of a subject, content, or participant in the audio data 102.In alternative embodiments, the metadata 108 may provide informationregarding context or content of the audio data 102, including a topic,time, date, or location etc.

The audio data 102 and the metadata 108 are provided to a speech-to-text(STT) server 104, which may employ any of a variety of method oftechniques for automatic speech recognition (ASR) to create an automatedspeech-to-text transcription 106 from the audio file. The transcriptionperformed by the STT server at 104 can exemplarily be a large-vocabularycontinuous speech recognition (LVCSR) and the audio data 102 provided tothe STT server 104 can alternatively be a previously recorded audio fileor can be streaming audio data obtained from an ongoing communicationbetween two speakers. In an exemplary embodiment, the STT server 104 mayuse the received metadata 108 to select one or more models or techniquesfor producing the automated transcription. In a non-limiting example, anidentification of one of the speakers in the audio data can be used toselect a topical linguistic model based upon a content area associatedwith the speaker. Such content areas may be technological, customerservice, medical, legal, or other contextually based models. In additionto the transcription 106 from the STT server 104, STT server 104 mayalso output time stamps associated with particular transcriptionsegments, words, or phrases, and may also include a confidence score inthe automated transcription. The transcription 106 may also identifyhomogeneous speaker speech segments. Homogenous speech segments arethose segments of the transcription that have a high likelihood oforiginating from a single speaker. The speech segments may exemplarilycorrespond to phonemes, words, or sentences.

After the transcription 106 is created, both the audio data 102 and thetranscription 106 are used for a blind diarization at 110. Thediarization is characterized as blind as the identities of the speakers(e.g. agent, customer) are not known at this stage and therefore thediarization 110 merely discriminates between a first speaker (speaker 1)and a second speaker (speaker 2), or more. Additionally, in someembodiments, those segments for which, a speaker cannot be reliablydetermined may be labeled as being of an unknown speaker.

An embodiment of the blind diarization at 110 receives the mono audiodata 102 and the transcription 106 and begins with the assumption thatthere are two main speakers in the audio file. The homogeneous speakersegments from the transcription 106 are identified in the audio file.Then, long homogeneous speaker segments can be split into sub-segmentsif long silent intervals are found within a single segment. Thesub-segments are selected to avoid splitting the long speaker segmentswithin a word. The transcription 106 can provide context to whereindividual words start and end. After the audio file has been segmentedbased upon both the audio file 102 and the transcription 106, theidentified segments are clustered into speakers (e.g. speaker 1 andspeaker 2).

In an embodiment, the blind diarization uses voice activity detection(VAD) to segment the audio data 102 into utterances or short segments ofaudio data with a likelihood of emanating from a single speaker. In anembodiment, the VAD segments the audio data into utterances byidentifying segments of speech separated by segments of non-speech on aframe-by-frame basis. Context provided by the transcription 106 canimprove the distinction between speech and not speech segments. In theVAD an audio frame may be identified as speech or non-speech based upona plurality of heuristics or probabilities exemplarily based upon meanenergy, band energy, peakiness, residual energy or using the fasttranscription; however, it will be recognized that alternativeheuristics or probabilities may be used in alternative embodiments.

The blind diarization at 110 results in the homogenous speaker segmentsof the audio data (and the associated portion of the transcription 106)being tagged at 112 as being associated to a first speaker or a secondspeaker. As mentioned above, in some embodiments, more than two speakersmay be tagged, while in other embodiments, some segments may be taggedas “unknown.” It is to be understood that in some embodiments the audiodata may be diarized first and then transcribed, or transcribed firstand then diarized. In either embodiment, the audio data/transcriptionportions tagged at 112 are further processed by a more detaileddiarization at 114 to label the separated speakers.

The separation of spoken content into different speaker sides requiresthe additional information provided by the agent model, the customermodel, or both models in order to label which side of a conversation isthe agent and which is the customer. A linguistic agent model 116 can becreated using the transcripts, such as those produced by the SIT server104 depicted in FIG. 1, or in other embodiments, as disclosed hereinfrom a stored database of customer service interaction transcripts,exemplarily obtained from customer service interactions across afacility or organization. It is recognized that in alternativeembodiments, only transcripts from a single specific agent may beconsidered. A linguistic agent model identifies language and languagepatterns that are unique or highly correlated to the agent side of aconversation. In some embodiments, similar identification of languagecorrelated to the customer side of a conversation is identified andcomplied into the customer model 118. The combination of one or more ofthese linguistic models are then used in comparison to the segmentedtranscript to distinguish between the agent and the customer, such asafter a blind diarization.

When a customer service agent's speech is highly scripted, thelinguistic patients found in the script can be used to identify theagent side of a conversation. A script is usually defined as a longstretch of words which is employed by many agents and is dictated by thecompany (e.g. “ . . . customer services this is [name] speaking how canI help you . . . ”), Due to the linguistic properties of a conversation,it is rare to find a relatively long (e.g. five or more, seven or more,ten or more) stretch of words repeated over many conversations andacross agents. Therefore, if such a long stretch of words is repeatedlyidentified in one side of a conversation, then there is an increasedprobability that this represents a script that is being repeated by anagent in the course of business.

However, in order to be responsive to customer needs, the number ofactual scripts used by an organization is usually small, and agents arelikely to personalize or modify the script in order to make the scriptmore naturally lit into the conversation. Therefore, withoutsupplementation, reliance solely upon scripts may lead to inaccurateagent labeling as many conversations go unlabeled as no close enoughmatches to the scripts are found in either side of the conversations.

Therefore, in an embodiment, in addition to the identification and useof scripts in diarization, the agents' linguistic speech patterns can bedistinguished by the use of specific words, small phrases, orexpressions, (e.g. “sir”, “apologize”, “account number”, “let metransfer you”, “what I'll do is”, “let me see if”, or others). Shorterlinguistic elements such as these constitute an agent linguistic cloud,that may be correlated to agent speech, but may also have a higherchance to appear in the customer side of a conversation, either bychance, or due to error in the transcription, blind diarization, orspeaker separation.

In one embodiment, the difference between these two techniques can besummarized as while script analysis looks for specific sequences ofwords, the agent linguistic cloud approach looks more towards thespecific words used, and their frequency by one side in a conversation.A robust linguistic model uses both approaches in order to maximize theability of the model to discriminate between agent and customer speech.

At 114 the agent model 116, and in some embodiments a customer model118, are applied to the transcript clusters resulting from the speakertagging 112. It will be recognized that in embodiments, thetranscription and the blind diarization may occur in either order. Inembodiments wherein the transcription is performed first, thetranscription can assist in the blind diarization while embodimentswherein the blind diarization is performed first, this diarized audiocan facilities transcription. In any event, the agent diarization at 114is provided with clustered segments of transcription that has beendetermined to have ordinated from a single speaker. To these clustersthe agent model 116 and customer model 118 are applied and adetermination is made as to which of the models, the clusteredtranscriptions best match. As a result from this comparison, one side ofthe conversation is tagged as the agent at 120 while the other side istagged as the customer. In embodiments wherein only an agent model isused, the transcription data that is not selected as being morecorrelated to the agent model is tagged as being associated to thecustomer. After the transcription has been tagged between agent andcustomer speech at 120, this transcription can further be used inanalytics at 122 as the labeling of the diarized conversation canfacilitate more focused analysis, exemplarily on solely agent speech orsolely customer speech.

FIG. 2 is a diagram that depicts an embodiment of a method 200 creatingand using a linguistic model for labeling 200. The diagram of FIG. 2 canbe generally separated into two portions, a training portion 202 inwhich the agent linguistic model is created, and a labeling portion 204in which the agent linguistic model is applied to a diarizedconversation in order to identify the speakers in the conversation as anagent or a customer.

Starting with 202, a set of M recorded conversations are selected at206. The set of recorded conversations can be a predetermined number(e.g. 1,000), or can be a temporally determined number (e.g. theconversations recorded within the last week or 100 hours ofconversation), or a subset thereof. It is understood that these numbersare merely exemplary of the size of the set and not intended to belimiting. The recorded conversations may all be stored at a repositoryat a computer readable medium connected to a server and in communicationwith one or more computers in a network.

In embodiments, the set of recorded conversations may be furtherprocessed and reduced, exemplarily by performing an automated analysisof transcription quality. Exemplary embodiments of such automatedtechniques may include autocorrelation signal analysis. As previouslymentioned above, the speech to text server may also output a confidencescore in the transcription along with the transcription. In an exemplaryembodiment, only those transcriptions deemed to be of a particular highquality or high confidence are selected to be used at 206.

The selected set of recorded conversations are diarized and transcribedat 208. In embodiments, the transcription and diarization may beperformed in the manner as disclosed above with respect to FIG. 1, or ina manner similar to that as described. In an alternative embodiment,when the recorded conversations in the sets selected at 206 are actualcalls recorded and analyzed by a company or organization, thetranscription and diarization may already be performed as part of theoriginal use and analysis of the audio recording of the conversation andtherefore the diarization and transcription may be already stored in therepository with, or associated with, the audio file.

At 210, the results of the transcription and diarization at 208 areseparated into a plurality of text documents wherein each documentcontains the text transcript of one speaker side of a conversation.Therefore, due to the nature of conversation, the number of textdocuments 210 is larger than the number of audio files in the setselected at 206 as each audio fife will likely be split into two, if notmore, text documents. This results in a set of N single speaker textdocuments where N>M.

At 212 the text documents produced at 210 are analyzed to identifylinguistic patterns typically used by agents and/or patterns used bycustomers. This analysis can be performed using some type of heuristicsuch as, but not limited to, identifying repetitive long phrases thatare highly correlated to an agent side of a conversation. In oneembodiment, long scripted combinations of words are extracted. Scriptextraction and identification produces highly reliable results when aknown script segment is identified; however, script extraction andlabeling can result in many files being identified as unknown orindeterminate when insufficient matches to the extracted script text areidentified.

The script extraction can be performed by analyzing the text documentsfront 210 to identify lists of word strings of a predetermined length(e.g. five, seven, or ten words) and the frequency among the text fileswith which these word combinations appear. It will be understood thatadjustments to the word string length. When lower, will create a modelthat identifies more text files as being on an agent side, while longerword string lengths will increase the accuracy that the identified textfiles are spoken by agents.

In an exemplary embodiment; identification of a script or otherheuristics such as other repetitive words or phrases in a text file isindicative of a file having been spoken by an agent. As referencedabove, when a script or another heuristic can be identified in a textfile, this can produce a highly reliable identification of agent speech;however, such a technique is limited in that many other files of agentspeech may be missed. Therefore, in an embodiment, the model training at202 further selects only those text files that included a script andtherefore were highly likely to have been spoken by an agent for furtherprocessing as disclosed herein at 214 to create the more robust agentlinguistic model.

At 214, the linguistic model can be refilled and/or extended beyond thebasic script identification and furthermore in embodiments an agentlinguistic model and a customer linguistic model may be created. Thismay exemplarily be performed by using the basic script identification aslabeled training data tor any standard supervised learning classifier.In embodiments of the systems and methods as disclosed herein, the anentscripts extracted from 212 can be used to create an agent subset of thetext documents from 210. In such applications, the extracted scripts 212are applied to the text documents from 210 in order to identify a subsetof text documents that can be accurately known to be the agent side ofconversations. This application can be performed by representing each ofthe text documents as a long string produced by the concatenation of allthe words spoken by a speaker in the conversation, and the text documentis associated with the text document that represents the other speakerside of the conversation in a group. For each side of the conversation,all of the extracted scripts from 212 are iterated over the text filesin order to identify extracted scripts in the text files and a score isgiven to the identification of the script within the text fileindicating the closeness of the script to the text identified in thetext file. Each text file representing one side of the conversation isgiven a final score based upon the identified scripts in that text fileand the text files representing two halves of the conversation arecompared to one another to determine which half of the conversation hasa higher score which is indicative of the agent side of theconversation. If the difference between the two scores is above aminimal separation threshold, then the text file identified to be theagent side of the conversation based upon the script analysis is addedto the subset that may be used in the manner described below with thecreation of an agent linguistic cloud.

As described above, after the subset of text files that are highlylikely to be agent sides of conversations has been identified, thesubset can be analyzed in order to create an agent linguistic modelbased as a linguistic cloud of word frequencies. In exemplaryembodiments, the word frequencies in the linguistic cloud can beextended to joint distributions of word frequencies to capturefrequencies not only of particular words, but phrases or sequences ofwords. When used for speaker labeling, embodiments of the agentlinguistic model can result in fewer unidentified, unknown, orinconclusively labeled text files, but due to the nature of aconversation, transcript; or diarization, embodiments can have lessaccuracy than those identifications made using the extracted scriptmodel.

In addition to the use of scripts by a customer service agent, theagent's speech can be distinguished by the use of certain words, shortphrases, or expressions. Shorter elements, including, but not limitedto, unigrams, bigrams, and trigrams that are more correlated orprevalent in agent's side of conversation. By automatedly creating thesubset where the agent side has been identified in the conversation asdisclosed above, the unigrams, bigrams, and trigrams obtained from thissubset are more accurately known to come from the agent and thus cancapture increased variability in the agent sides of the conversation.

In an embodiment, unigrams, bigrams, ingrains, words, or phrases thatare more prominent to the agent are extracted similar to the manner asdescribed above with respect to the script extractions. In anembodiment, those unigrams, bigrams, trigrams, words, or phrases thatboth occur frequently in the agent sides of the subset and appear morefrequently in the agent sides of the subset than the correspondingcustomer sides of the conversation by at least a predetermined amount,may be added to the agent linguistic cloud model. Once all of theelements for the agent linguistic cloud model have been extracted, theseelements, in addition to the previously extracted scripts, are allwritten in a text the as the agent linguistic model that can be used inthe labeling portion 204 as shown in FIG. 2. A similar process may beused to create the customer linguistic model.

At 204 the created agent linguistic model which may contain elements ofboth the script and cloud techniques (and in embodiments, the customerlinguistic model) are applied to a new call in order to diarize andlabel between an agent ands caller in the new call. In 204 a new call216 is received and recorded or otherwise transformed into an audiofile. It is to be recognized that embodiments, the labeling of 204 canbe performed in real time or near-real time as the conversation istaking place, or may occur after the fact when the completedconversation is stored as an audio file. The new call 216 is diarizedand transcribed at 218 which may occur in a similar manner as describedabove with respect to FIG. 1 and particularly 108, 110, and 112 atFIG. 1. As the result of such a blind diarization as exemplarilydescribed above, the system and method still requires to identify whichspeaker is the agent and which speaker is the customer. This isperformed in an agent diarization at 220 by applying the agentlinguistic model and customer linguistic model created in 202 to thediarized transcript. In application at 220, the agent linguistic modelis applied to both of the sides of the conversation and counted orweighted based upon the number of language specific patterns from theagent linguistic model are identified in each of the conversation halvesidentified as a first speaker and a second speaker. The conversationhalf with the higher score is identified as the agent and the otherspeaker of the other conversation half is identified as the customer.

It will be understood that in some embodiments of methods as disclosedherein, an agent linguistic model may be used in conjunction with otheragent models, exemplarily an agent acoustical model that models specificacoustical traits attributed to a specific, agent known to be one of thespeakers in a conversation. Examples or acoustical voiceprint models areexemplarily disclosed in U.S. Provisional Patent Application No.61/729,064 filed on Nov. 21, 2012, which is hereby incorporated byreference in its entirety. In some embodiments, linguistic models andacoustic models may be applied in an “and” fashion or an “or” fashion,while in still further embodiments, the different models are performedin a particular sequence in order to maximize the advantages of bothmodels.

In exemplary embodiments of combined use of a linguistic model and anacoustic voiceprint model, the application of the models may beperformed in parallel, or in conjunction. If the models are performed inparallel, the resulting speaker diarization and labeling from each ofthe models can be compared before making a final determination on thelabeling. In such an exemplary embodiment, if both models agree on thespeaker label, then that label is used, while if the separate modelsdisagree, then further evaluation or analysis may be undertaken in orderto determine which model is more reliable or more likely to be correctbased upon further context of the audio data. Such an exemplaryembodiment may offer the advantages of both acoustic and linguisticmodeling and speaker separation techniques. In exemplary embodiments,linguistic models may be better at making a distinction between agentspeech and customer speech, while acoustic models may be better atdiscriminating between speakers in a specific audio file.

In a still further embodiment, the combination of both an acousticvoiceprint model and a linguistic model can help to identify errors inthe blind diarization or the speaker separation phases, exemplarily byhighlighting portions of the audio data and transcription within whichthe two models disagree and for facilitating a more detailed analysis inthose areas in order to arrive at the correct diarization in speakerlabeling. Similarly, the use of an additional acoustic model may providea backup for instance wherein a linguistic model is not available. Suchan exemplary embodiment may occur when analyzing audio data of anunknown topic or before a linguistic model can be created, such asdescribed above with respect to FIG. 2.

In still further embodiments, the use of a combination of acoustic andlinguistic models may help in the identification and separation ofspeakers in audio data that contain more than two speakers, exemplarily,one customer service agent and two customers; two agents, and onecustomer; or an agent, a customer, and an automated recording. Asmentioned above, embodiments of a linguistic model may have strength indiscriminating between agent speech and customer speech while anacoustic model may better distinguish between two similar speakers,exemplarily between two agents or two customers, or an agent and arecorded voice message.

This written description uses examples to disclose the invention,including the best mode, and also to enable any person skilled in theart to make and use the invention. The patentable scope of the inventionis defined by the claims, and may include other examples that occur tothose skilled in the art. Such other examples are intended to be withinthe scope of the claims if they have structural elements that do notdiffer from the literal language of the claims, or if they includeequivalent structural elements with insubstantial differences from theliteral languages of the claims.

What is claimed is:
 1. A method of diarization, the method comprising:receiving a set of textual transcripts from a transcription server and aset of audio files associated with the set of textual transcripts froman audio database server; performing a blind diarization on the set oftextual transcripts and the set of audio files to segment and clusterthe textual transcripts into a plurality of textual speaker clusters,wherein the number of textual speaker clusters is at least equal to anumber of speakers in the textual transcript; automatedly applying atleast one heuristic to the textual speaker clusters with a processor toselect textual speaker clusters likely to be associated with anidentified group of speakers; analyzing the selected textual speakerclusters with the processor to create at least one linguistic model;applying the linguistic model to transcribed audio data with theprocessor to label a portion of the transcribed audio data as havingbeen spoken by the identified group of speakers; saving the at least onelinguistic model to a linguistic database server and associating it withthe labeled speaker; with the processor, receiving a new textualtranscript from the transcription server and a new audio file associatedwith the new textual transcript from the audio database server;receiving the at least one linguistic model from the linguistic databaseserver; receiving at least one acoustic voiceprint associated with aspecific speaker from a voiceprint database server; applying thereceived at least one linguistic model from the linguistic databaseserver to the new audio file transcript from an audio source to performdiarization of the new audio file by blind diarizing the new audio fileand new textual transcript, comparing each new textual speaker clusterto the at least one linguistic model, and labeling each textual speakercluster as belonging to a customer service agent or belonging to acustomer, comparing each audio speaker segment to the at least oneacoustic voiceprint, and labeling each audio speaker segment asbelonging to a known speaker or belonging to an unknown speaker; whenone of the audio speaker segments is labeled as belonging to a knownspeaker, selecting and transcribing the labeled audio speaker segmentswith the transcription server; comparing the selected transcribedlabeled audio speaker segments to the textual speaker clusters labeledas belonging to a customer service agent; and when the comparedtranscribed segments and clusters are each labeled as belonging to aknown speaker and a customer service agent, keeping the current labels,otherwise relabeling the textual speaker cluster as belonging to anunknown speaker.
 2. The method of claim 1, wherein the identified groupof speakers are customer service agents and the audio data is audio dataof a customer service interaction between at least one customer serviceagent and at least one customer.
 3. The method of claim 1, wherein thespecific speaker is a specific customer service agent.
 4. The method ofclaim 1, wherein the at least one heuristic is detection of a scriptassociated with the identified group of speakers.
 5. The method of claim1, wherein the analysis of the selected textual speaker clustersincludes determining word use frequencies for words in the selectedtextual speaker clusters with the processor, determining word usefrequencies for words in the non-selected textual speaker clusters withthe processor, and comparing the word use frequencies for words in theselected textual speaker clusters to the word use frequencies for wordsin the non-selected textual speaker clusters with the processor toidentify a plurality of discriminating words for use in the at least onelinguistic model.
 6. The method of claim 1, wherein the analysis of theselected textual speaker clusters includes receiving a plurality ofscripts associated with the identified group of speakers, comparing theplurality of scripts to the selected textual speaker clusters, comparingthe plurality of scripts of non-selected textual speaker clusters,determining a correlation score between each of the textual speakerclusters and the plurality of scripts, identifying the group with thegreatest correlation score for use in the at least one linguistic model.7. The method of claim 6, further comprising: calculating a differencebetween the word use frequencies for each word in the selected textualspeaker clusters and the non-selected textual speaker clusters; andcomparing the difference to a predetermined selection threshold, whereinif the difference is greater than the predetermined selection threshold,the word is identified as a discriminating word.
 8. The method of claim1, wherein the textual speaker clusters are associated in groups of atleast two, wherein the group of at least two includes a textual speakercluster originating from the identified group of speakers and at leastone textual speaker cluster originating from an other speaker, andwherein the non-selected textual speaker clusters are assumed to haveoriginated from an other speaker.
 9. The method of claim 1, wherein theat least one acoustic voiceprint is a set of acoustic voiceprints foreach specific customer service agent saved in the acoustic voiceprintdatabase server.
 10. The method of claim 9, the method furthercomprising: receiving the set of acoustic voiceprints from the acousticvoiceprint database server; applying the received at least onelinguistic model from the linguistic database server to the new audiofile transcript from an audio source to perform diarization of the newaudio file by blind diarizing the new audio file and new textualtranscript; comparing each new textual speaker cluster to the at leastone linguistic model, and labeling each textual speaker cluster asbelonging to a customer service agent or belonging to a customer;comparing each audio speaker segment to the set of acoustic voiceprints;determining which audio speaker segments match one of the acousticvoiceprints; and labeling those audio speaker segments as belonging tothe known speaker.
 11. A system for diarization and labeling of audiodata, the system comprising: An audio database server comprising aplurality of audio files; a transcription server that transcribes theaudio files of the plurality of audio files into textual transcripts; aprocessor that receives a set of textual transcripts from thetranscription serve and a set of audio files associated with the set oftextual transcripts from the audio database server, performs a blinddiarization of the set of textual transcripts and the set of audio filesto segment and cluster the textual transcripts into a plurality oftextual speaker clusters, segment and cluster the audio files into aplurality of audio speaker segments, wherein the number of textualspeaker clusters and the audio speaker segments are each at least equalto a number of speakers in the textual transcript, automatedly appliesat least one heuristic to the textual speaker clusters to select atleast one of the textual speaker cluster as being associated to anidentified group of speakers, and analyzes the selected transcripts tocreate at least one linguistic model indicative of the identified groupof speakers; a linguistic database server that stores the at least onelinguistic model an acoustic voiceprint database server that stores theat least one acoustic voiceprint from a known speaker; and an audiosource that provides new transcribed audio data to the processor;wherein the processor, receives a new textual transcript from thetranscription server and a new audio file associated with the newtextual transcript from the audio database server, receives the at leastone linguistic model from the linguistic database server, receives atleast one acoustic voiceprint associated with a specific speaker from avoiceprint database server, applies the at least one linguistic modelfrom the linguistic database server to the new audio file transcriptfrom an audio source to perform diarization of the new audio file byblind diarizing the new audio file and new textual transcript, compareseach new textual speaker cluster to the at least one linguistic model,and labels each textual speaker cluster as belonging to a customerservice agent or belonging to a customer, compares each audio speakersegment to the at least one acoustic voiceprint, and labels each audiospeaker segment as belonging to a known speaker or belonging to anunknown speaker, when one of the audio speaker segments is labeled asbelonging to a known speaker, selects and transcribes the labeled audiospeaker segments with the transcription server, compares the selectedtranscribed labeled audio speaker segments to the textual speakerclusters labeled as belonging to a customer service agent; and based onthe comparison, when the compared transcribed segments and clusters areeach labeled as belonging to a known speaker and a customer serviceagent, keep the current labels, otherwise relabel the textual speakercluster as belonging to an unknown speaker.
 12. The system of claim 11,wherein the identified group of speakers are customer service agents andeach of the audio files is of a customer service interaction between atleast one customer service agent and at least one customer.
 13. Thesystem of claim 11, wherein the specific speaker is a specific customerservice agent.
 14. The system of claim 11, wherein the at least oneheuristic is detection of a script associated with the identified groupof speakers.
 15. The system of claim 11, wherein the analysis of theselected textual speaker clusters includes determining word usefrequencies for words in the selected textual speaker clusters with theprocessor, determining word use frequencies for words in thenon-selected textual speaker clusters with the processor, and comparingthe word use frequencies for words in the selected textual speakerclusters to the word use frequencies for words in the non-selectedtextual speaker clusters with the processor to identify a plurality ofdiscriminating words for use in the at least one linguistic model. 16.The system of claim 11, further comprising: a processor that calculatesa difference between the word use frequencies for each word in theselected textual speaker clusters and the non-selected textual speakerclusters, and compares the difference to a predetermined selectionthreshold, wherein if the difference is greater than the predeterminedselection threshold, the word is identified as a discriminating word,wherein the analysis of the selected textual speaker clusters includesthe processor receiving a plurality of scripts associated with theidentified group of speakers, compares the plurality of scripts to theselected textual speaker clusters, compares the plurality of scripts ofnon-selected textual speaker clusters, determines a correlation scorebetween each of the textual speaker clusters and the plurality ofscripts, identifies the group with the greatest correlation score foruse in the at least one linguistic model.
 17. The method of claim 11,wherein the at least one acoustic voiceprint is a set of acousticvoiceprints for each specific customer service agent saved in theacoustic voiceprint database server.
 18. The method of claim 17, themethod further comprising: the processor further receives the set ofacoustic voiceprints from the acoustic voiceprint database server,applies the received at least one linguistic model from the linguisticdatabase server to the new audio file transcript from an audio source toperform diarization of the new audio file by blind diarizing the newaudio file and new textual transcript; compares each new textual speakercluster to the at least one linguistic model, and labels each textualspeaker cluster as belonging to a customer service agent or belonging toa customer; compares each audio speaker segment to the set of acousticvoiceprints; determines which audio speaker segments match one of theacoustic voiceprints; and labels those audio speaker segments asbelonging to the known speaker.